Repository navigation
fix(envs): confine benchmark task output paths - #309
Yifan Yang (Yif-Yang) merged 1 commit into
Conversation
|
Reviewed Thank you for addressing #306. The focused 59-case selection verifies early rejection of unsafe identifiers, unchanged normal layouts, mapping-collision and directory-link checks, confined legacy reads, and all four normal benchmark writer/reader controls. A real LiveMath rollout writes the mapped trajectory and slow-update reads it. This closes the reviewed task-output scope, not broader staging/release issues or a claim of complete filesystem isolation. Each step was re-simulated on the then-current main and matched the audited tree; the original PR author is retained as commit author. The final combined main ( |
2 participants
Summary
Extend the task-output confinement introduced in #264 to DocVQA, SearchQA, OfficeQA, and LiveMathematicianBench. The benchmark rollouts share one path resolver; safe identifiers keep their existing directory names, while LiveMath identifiers such as
202602:12map to stable directory segments that work on Windows.Slow-update trajectory reads use the same mapping and retain support for legacy POSIX LiveMath directories after checking containment. Regression tests cover rejected identifiers, symlink escapes, mapped rollout output, and both mapped and legacy trajectory reads.
Fixes #306.
Testing
python -m pytest -q tests/test_benchmark_task_output_paths.py tests/test_spreadsheetbench_task_id_confinement.py tests/test_slow_update_robustness.py— 34 passed.python -m pytest -q— 1,723 passed, 11 skipped, 359 subtests passed.uvx ruff check skillopt/envs/task_output.py skillopt/optimizer/slow_update.py tests/test_benchmark_task_output_paths.py— passed.git diff --check— passed.The test runs above were completed on October 7. Before submission, the October 8 check confirmed that upstream
mainremains at343db229dbd5ddaf9df6b1d5540d8bcdb2604d5cand that all eight changed files match the tested patch exactly.Existing Ruff findings in the full rollout files reproduce on the unchanged upstream baseline and are outside this change.